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WHAT IS CLAIMED IS: 



nJ 



A method fW recovering 3D scene structure and camera motion from image data 
obtained from a multi^mage sequence, wherein a reference image of the sequence is taken by 
5 a camera at a reference perspective and one or more successive images of the sequence are 
taken at one or more successive different perspectives by translating and/or rotating the 
camera, the method comprising the steps of: 

(a) determining image\data shifts for each successive image with respect to the 
reference image; the shifts being derived from the camera translation and/or rotation from the 

10 reference perspective to the successive different perspectives; 

(b) constructing a shift dafa matrix that incorporates the image data shifts for each 

image; 

factor matrices from the shift data matrix using SVD, one 
g to the 3D structure and the other rank-3 factor matrix 
15 corresponding tothe camera moti m; 

(d) recovering the 3D structure from the 3D structure matrix by solving a linear 
equation; and 

(e) recovering the earner \ motion from the camera motion matrix using the recovered 
3D structure. 

20 

2. The method of claim 1 , Wherein the image data is one or more selected from the 



(c) calculating two rank-3 
rank-3 factor matrix correspondii 
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group consisting of points, lines and intensities. 



3. The method of claim 1 , wherein the step of determining image data shifts includes 
initially recovering and compensating for camera rotation. 



10 



15 



20 



The method of claim l\ wherein step (b) comprises: 

computing H and D CH , Where H is a (N iot - 3)x N tot matrix and D CH = C' m D H T 
and N XoX = 2Np + 2 Ni + Nx where Np, Ni, and Nx equal the number of points, lines and 
intensities, respectively, and C is a (N, -1) x (N, -1) matrix with C w = S ir + 1 where C, r is a 
constant table of values for all images and S ir is an operator equal to 1 if / = /' and 0 if i * i r , 
and D = [S o) L A co x a] , H T is the transpose of the H matrix, where H is a matrix defined 
such that H T is an identity matrix Lid annihilates W x9 ¥ v ,% where = [yJ <a L YL co^l] 
and similarly for the y and z components where, co x and co L are constant weights that the user 



sets, and where = 



> t -(xxA 



VI/ = 

T Z.v _ 



'{p y -(yxA)}' 

.{Pr(yxA))_ 



.{V(zxA)}_ 



where ^is 



the unit normal to the plane contair ing the line and the camera center at the reference image 
and where x, y, z are unit vectors ii l the x, y and z directions, and where 



where q=(x,y) is the image position of the 





'{•*"(«)}" 










>,»}" 


f 5 ! 


>f(4. 


9 T y 






P r 3 


.frfto}. 



tracked point in the reference image and 



the three point rotational flows ( x, ;/), r ^ (x, y\ r ^ (x, j>) are defined by 



P\* Ik 

Y and where 

¥ k = -{v/ • r (i) (p)}, ¥ Iv = -{w ■ r (2) ( ))}, Y b = -{w • r (3) (p)} , and where p gives the image 
coordinates of the pixel positions, aid 



25 where A is an N f x 2N L matrix wt ere each row corresponds to a different image i and 

equals [{p^ * <SA' } r \p l • SA' } T wher i and P L are unit 3 -vectors projecting onto two 
directions A, x (i x A, )and z x A, vmich we refer to respectively as the upper and lower 
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directions, where SA l is th&line flow SA] s A) - A? , and z is the unit vector in the z 
direction and where S is a A/\x 2N P matrix where each row corresponds to a different image 

i and equals ^s' x f {s' v } T ] , whete s' m s= q^ - q^ and denotes the image displacement for the 
m-th tracked point and where A Is a N f x N x matrix, where each row corresponds to an 

image i and equals {aI' f where W is the change in image intensity with respect to the 

reference image and where ./' denotes the z-th intensity image and where V n = V (p n ) denotes 

the image intensity at the /2-th pixel position in/' and where the notation {V} is used to 
denote a vector with elements given W the V a . 



5. The method of claim 1, whereinVtep (c) comprises: 

computing the best rank-3 factorisation of D CH « M^S ( ^ T where are rank 

3 matrices corresponding respectively to motion and structure, using SVD. 



Q 



15 



20 



25 



6. The method of claim 1, wherein step 1(d) comprises: 

eliminating structure unknowns Q z , B z and z _1 from the O a to get 3N tot linear constraints 



on the U and CI using the linear equation, [< 
and Cl are unknown 3x3 matrices, and O anc 



SVD, where the total translational flow vectq 
for the y and z components, and 



O v <dJ=H 7 '5 (3) [/ + [ x P jc *F v Y z ]q, where U 
*F represent total translational and rotational 



flow vectors respectively, and solving these constraints with 0(N tot ) computations using the 



s are defined by O x = 



and similarly 





















U. V B L } 







where A is the normal to a first plane that 



passes through the center of projection and/the imaged line in the reference image, and the 
normal to a second plane, B is defined by requiring B • A = 0 and B • Q = - 1 for any point Q 



the 3D line L, and where B u =B*P a and 
B, and where 
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1(0} 



[ ! °n<i> 



and where 



where Q is the 3d coordinate for a 3D tracked point corresponding to an image pixel 
in the reference image, and Z n is me depth from the camera to the 3D point imaged at the n- 
th pixel along the cameras optical a?qs; and 

recovering the structure unknowns Q z , B z and Z" 1 from 
\® x O x O x ] = H r 5 ( % + [^ W v %lfl, gi\en U and Q . 



7. The method of claim 1, wherein steppe) comprises: 
using and 



recovering the rotations G}^&\^& l z from 



i/ 



f l^z^H 1 to recover the translations, and 



^ + + = C" X D: -(c^fe}0 r + {r>; + {r 2 }0[)J , wherein C 



is a 



constant (Nj-1) x (N,-l) matrix with C /V ^ 8 r , + 1 and 7 represents the translation. 

8 . A method for recovering 3D scene structure and camera motion from image data 
obtained from a multi-image sequence, wherein a reference image of the sequence is taken by 
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a camera at a reference perspective and one or more successive images of the sequence are 
taken at one or more successive different perspectives by translating and/or rotating the 
camera, the method comprising the steps of: 

(a) initially recovering and compensating for camera rotation after warping the 
5 reference image by the previously estimated translational displacements; 

(a) determining image data shifts for each successive image with respect to the 
reference image; the shifts beingWerived from the camera translation and/or rotation from the 
reference perspective to the successive different perspectives; 

(b) constructing a shift dat\ matrix that incorporates the image data shifts for each 

10 image; 

(c) modifying the image shifts by multiplying the image shifts 5A\ 9 s' m ,M' n and 
w redefining these quantities to include the denominator factors 1 - Q~ X T Z for points, 1 - Z~ X T Z 
S for intensities and l-B-T for lines where s' m = q^ - q° and denotes the image displacement 
il for the m-th tracked point, where Al' n is the change in image intensity and where /' denotes 

15 the i-th intensity image and wher//^ = /' (p n ) denotes the image intensity at the n-th pixel 
position in/' and where SA\ represents the line flow 

(d) calculating two ranlc-3 factor matrices from the modified shift data matrix using 
SVD 5 one rank-3 factor matrix; corresponding to the 3D structure and the other rank-3 factor 
matrix corresponding to the camera motion; 

20 (e) recovering the 3D|structure from the 3D structure matrix by solving a linear 
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equationjWd 

(f) recovering the camera motion from the camera motion matrix using the recovered 
3D structure. 



5 9. The method of claim 8 ^wherein the image data is one or more selected from the 
group consisting of points, lines and intensities. 

if 



10. The method of claim 8, wherein step (b) comprises: 

computing H and D CH , where HVs a (N tot - 3)x N tot matrix and D CH = C' m D H T 
and N tot = 2Np + 2 N£ + Nx, where Np, Ml, and Nx equals the number of points, lines and 
intensities, respectively, and C is a (N, -l)\x (N, -1) matrix with C ir = S ir + 1 where C ie is a 
constant table of values for all images and <L is an operator equal to 1 if / = /' and 0 if / * /' 
and D = [S co L A o) { A], H t is the transpose of the H matrix, where H is a matrix defined 
such that H T is an identity matrix and annihilktes ¥ v ,¥ 2 where = [*Fj fi> L ¥£ ^J,] 
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25 



and similarly for the y and z components and 

■^•(xxA)}" 



whereby, and to L are constant weights that the 



user sets, and where = 



,{V(xx A )}_ 



5 T L 



>,-(yxA)}" 
.(v(yxA)}. 



xp = 



'{V(z*A)}' 
{P £ -(SxA)} 



where 



i4, is the unit normal to the plane containing fr e line L and the camera center at the reference 
image, and where x, y, z are unit vectors in the x, y and z directions, and where 



'{•%)}' 


5 T y 


>i 2, (?)}" 






»»1 


.{<f(4. 









20 tracked point in the reference image and 



where q=(x,y) is the image position of the 



the three point rotational flows r (l) (xj)/ (2 ]f(xj),r (3) (xj) are defined by 

[^"^ , "^ < "]-[(:^)).(r').(;')] A k 

L J and where 

*F U = -{v/ • r (1) (p)}, ¥ lv * -{v/ • r (2) (p)}, ^{ = -{v/ • r (3) (p)}, and where p gives the image 
coordinates of the pixel positions, andy 
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where A is an $)s$2N L matrix where each row corresponds to a different image i and 

equals [{p^ • SA l f {p^SA/ } r where P^ and P L are a unit 3-vector projecting onto two 
directions A, x (z x A, )andsz x A, which we refer to respectively as the upper and lower 
directions, where SA l is the line flow SA] = A] - A? , and z is the unit vector in the z 
direction and where 5 is a N i x\N p matrix where each row corresponds to a different image 

/ and equals [{s^ } r {s' v f ] , where V = q' m - q° m and denotes the image displacement for the 
m-th tracked point and where A is Af, x N x matrix, where each row corresponds to an 

image i and equals {a! ! ) T where AAis the change in image intensity with respect to the 

reference image and where /' denoted the i-th intensity image and where I' n = I'(p n ) denotes 

the image intensity at the n-th pixel position in/' and where the notation { V}is used to 
denote a vector with elements given bM the V a . 



1 1 . The method of claim 8, wherein 
computing the best rank-3 factori: 



step (d) comprises: 

izationof D CH *M (3) S (3)r where M (3) ,S (3) are rank 



3 matrices corresponding respectively to motion and structure, using SVD. 



12. The method of claim 8, wherein step (e) comprises: 



eliminating structure unknowns Q z , B 2 and Z 1 from the <D 0 to get 3N tot linear constraints 

on the U and Q using the linear equation, [o, O v <D Z ] = U T S {3) U + ^¥ x % % ]ft , where U 

and Q are unknown 3x3 matrices, and <E> and ¥ represent total translational and rotational 
flow vectors respectively, and solving these constraints with 0(N tot ) computations using the 

and similarly 



SVD, where the total translational flow vectors are defined by O x = 
for the y and z components, and 

where A is the normal to a first plane that 

passes through the center of projection and the imaged line in the reference image, and the 
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normal to a secoft^ plane, B is defined by requiring B • A = 0 and B * Q = - 1 for any point Q 
on the 3D line L, 

and where B v = B • A and B L = B • ? L are the upper and lower components of B, and where 



fei- 
lo}. 



10 



15 



and where 



where is the 3d coordinate for a tracked 3D point in the reference image, and Z n is 
the depth from the camera to the 3D point imaged at the n-th pixel along the cameras optical 



axis; and 



recovering the structure unknowns Q z , B z and Z 1 from 
[o x <D X oJ = H 7 5 { % + [^ Y v ¥ 2 )Q , given £/ and Q . 



13. The method of claim 8, wherein step (f) comprises 



using 5 ( %«[o x O v <D z ] 



and 



D c// » C"^{rJo r H T +C"^{r v }O v r H T +C~^{7;}o[H T to recover the translations, and 



recovering the rotations o) l x ,6) l v ,co 



20 «)X+<^+®^- = ^D 



from 



-^C^({rJo r + {7; v }o; +{rJo[)J , wherein Cis ; 



constant (Nj-1) x (N,-l) matrix with C |T = S ir + 1 and T represents the translation. 
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